You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-config [6/13]: decide when it is safe to apply GPU settings - #3252
The changes add GPU configuration parsing and matching, state-file storage, driver readiness checks, and NVML-backed setting application. A decision flow checks driver and cluster evidence, then records whether configuration was applied or skipped. The new tests cover parsing, matching, state operations, NVML calls, and decision outcomes.
Merge Risk: Low · up to d71d0
This change adds the GPU configuration decision flow but does not use it yet, so there is no immediate production impact. One error path returns a "driver reloaded" outcome alongside a transient read error, and a few tests are looser than they should be. Both are worth fixing before the flow is wired in, but neither blocks merging.
Security architecture risk: Minimal · up to d71d0
The change warrants design review because it controls when GPU settings may change. It is not activated in running deployments, and the inspected paths establish no introduced security or failure-containment defect.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — When integrated with sufficient device privileges, one decision can configure every selected GPU visible to its NVML session, affecting workloads sharing those GPUs rather than only one pod or tenant. Current deployed reachability was not identified.
Trust Boundaries and Controls
observed — The gate treats prior gate passes, toolkit readiness, advertised GPU capacity, relevant ResourceSlices, and GPU pods as evidence against changing settings. Required read errors stop application. Applier exclusion is scoped by namespace and controller DaemonSet name; unresolved claim references are conservatively classified as GPU-related.
Resilience and Maintainability Implications
observed — Driver readiness is checked before opening NVML, and load identity is rechecked after device enumeration. Unsupported or unreadable virtualization modes produce a recorded skip without changing settings.
Hardening Proposals
proposed — Before activation, verify that production wiring supplies trusted node and namespace identity, implements the stated read-freshness contracts, and makes allocator-release gates participate in the same lock protocol.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Other comments: 2
internal/gpuconfig/decision/decide.go: 1
187-189: Minor · Functional Correctness · Quick win
Return None when loadChanged fails.
When d.LoadID() returns an error, loadChanged returns (false, ErrTransient-wrapped err). Line 188 and line 381 still return Result{Outcome: LoadChanged} together with that error. The Outcome doc says None is "the outcome returned with an error". A caller that reads Outcome before err treats a transient read failure as a driver reload and rebinds. Return a zero Result when err != nil.
669-669: Minor · Functional Correctness · Quick win
Reject unexpected errors in success cases.
If the document-application case returns Result{} and an unexpected error before writing state, this branch accepts None, unchanged state and closed NVML. The subtest then passes without checking Applied.
Enter this branch only when the case expects an error. Otherwise, assert require.NoError(t, err) before checking the outcome.
As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”
This test starts with l.base == nil. It still passes if Read incorrectly retains the base after a later read fails.
First complete a successful read. Then corrupt the file and repeat the read. Assert that WriteGatePass fails and leaves the corrupt bytes unchanged. This covers the invalidation contract; the supplied implementation already satisfies it.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
Test load-ID lifetime instead of only repeating the inode lookup.
The expected value uses the same os.Stat and Stat_t.Ino operations as Read. An implementation that caches the first successful ID for a path would pass this test.
Assert that repeated reads return the same ID. Then rename the original directory aside, create a new directory at the original path, and assert that the ID changes. Keeping the original directory prevents inode reuse in this fixture.
As per path instructions, “Flag tests that mirror the implementation instead of checking behavior.”
Check that evidence-based skips prevent setting writes.
Initialized == false also holds after NVML opens and shuts down. These cases supply no configuration document, so they cannot detect accidental application of configured settings before returning Skipped.
Supply a document with docYAML and assert that e.log.Writes stays empty in the skip cases. Keep the existing outcome and state assertions.
As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”
Reviewing files that changed from the base of the PR and between 246a9ca and d71d0db.
Files ignored due to path filters: 39
go.sum is excluded by !**/*.sum, !go.sum
vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 26
go.mod
internal/gpuconfig/apply/apply.go
internal/gpuconfig/apply/apply_test.go
internal/gpuconfig/apply/fake/fake.go
internal/gpuconfig/apply/nvml.go
internal/gpuconfig/apply/nvmllib/nvmllib.go
internal/gpuconfig/apply/nvmllib/nvmllib_test.go
internal/gpuconfig/config/document.go
internal/gpuconfig/config/document_test.go
internal/gpuconfig/config/match.go
internal/gpuconfig/config/match_test.go
internal/gpuconfig/config/schema.go
internal/gpuconfig/decision/decide.go
internal/gpuconfig/decision/decide_test.go
internal/gpuconfig/decision/evidence.go
internal/gpuconfig/decision/helpers_test.go
internal/gpuconfig/decision/pods.go
internal/gpuconfig/loadid/driver.go
internal/gpuconfig/loadid/loadid.go
internal/gpuconfig/loadid/loadid_test.go
internal/gpuconfig/statefile/file.go
internal/gpuconfig/statefile/file_test.go
internal/gpuconfig/statefile/state.go
internal/gpuconfig/statefile/state_test.go
internal/gpuconfig/status/sync.go
internal/gpuconfig/status/sync_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.
henry118
changed the title
gpu-config: decide when it is safe to apply GPU settings
gpu-config [6/13]: decide when it is safe to apply GPU settings
Nov 19, 2026
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3246 (6/13). Depends on: #3247, #3250, #3251. Review only the last commit.
Adds
internal/gpuconfig/decision: the once-per-driver-load decision toapply or skip. This is the safety-critical part. Not used yet.